COMPUTING HITTING TIMES VIA FLUID APPROXIMATION: 
APPLICATION TO THE COUPON COLLECTOR PROBLEM 

NICOLAS CAST 

^— ^ Abstract. In this paper, we show how to use stochastic approximation to 

f"^ compute hitting time of a stochastic process, based on the study of the time 

^vj for a fluid approximation of this process to be at distance 1/A'^ of its fixed 

^_^ point. 

r^ This approach is developed to study a generalized version of the coupon 

h— -J collector problem. The system is composed by N independent identical Markov 

chains. At each time step, one Markov chain is picked at random and performs 
one transition. We show that the time at which all chains have hit the same 
state is bounded by ciNlogN + C2A'^loglog Af + 0{N) where ci and C2 are 
two constants depending on eigenvalues of the Markov chain. 
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1. Introduction 



H The coupon collector is a classical problem in probability theory. There are N 

'— ^ types of coupons. Coupon are collected at random with replacement. The goal is to 

^^ compute the number of coupons to be collected to have at least one coupon of each 

^ kind. This simple problem has a simple answer and a simple proof: on average, 

I/-) one has to buy N/N + {N - 1)/N + • • • + 1/N « TVlogiV coupons to complete 

OO a collection: if you already have k different types of coupons, it takes in average 

^n (iV — k)/N to get a coupon of a new type. 

Because of its simplicity, this problem has many applications, especially in coni- 



es**' puter science where it often serves as a basic tool for computing the completion 

^— ^ time of randomized algorithm [11, 8]. Many variants of it have been studied during 

, the years. For example, the time needed to complete a T collections of the same N 

•^- coupons is shown to be iV(log N + {T-1) log log N + 0(1)) in [l:'), 12] . The time to 

complete the first collection is TV log A^. However, the time to complete each next 

collection is only N log log N. 

%^ However, even a slight modification such has obtaining T different collections 

instead of one leads to much more complicated proofs. The approach taken in this 

paper aims at being more general but also at giving a new insight on the relation 

between hitting time and stochastic approximation. 

Contributions. We develop an approach based on stochastic approximation to 
compute the hitting time of a stochastic process that has an absorbing state. The 
system is composed of N identical Markov chains that have an absorbing state 0. 
At each time step, one chain is picked at random and performs one transition. Our 
goal is to compute the number of steps until which all Markov chains are in their 
absorbing state. The coupon collector problem is a particular case of this problem 
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by considering N deterministic Markov chains that have state 1 (no coupon of that 
type has been collected) or (at least one coupon of type i has been collected) . 

Using a classical stochastic approximation approach, like [ ], one can show that 
if N is large, the proportion of Markov chains that are in a given states can be ap- 
proximated by a linear ordinary differential equation (ODE) m — mQ. This ODE 
has a unique fixed point to which all trajectories converge exponentially fast, corre- 
sponding to a state where all chains are in state 0. However, this approximation is 
not enough accurate to bound the hitting time of the stochastic process. The time 
for this ODE to reach its equilibrium is infinite whereas the expected time for the 
stochastic system to hit this equilibrium is finite. 

In this paper, we establish a relation between the expected time E [Tjv] for all the 
chains to be completed and the time ijv for the ODE to be at distance 1/iV of its 
equilibrium point. The main results of this paper are Theorem 1 and Theorem 2. 
We first show that T/v is bounded by N ■ t^. Using this result, we derive the 
existence of constants ci , C2 that depend on the spectral properties of the original 
Markov chain such that 

E [Tn] < NtN + 0{N) 

< ciN log N + C2N loglog N + 0{N). 

Applied to the time to complete T collections, the allows to derive directly the 
results of [ ]. 

We also study two particular cases for which we have simple closed-form bounds 
for T/v. In the more general case, if we only know that the expected hitting time of 
one Markov chain is bounded by T starting from its initial state, then Tjv is only 
bounded by N'^T. However, if the expected hitting for a Markov chain is bounded 
by T independently of its initial state, then we show that E [T^v] < NT log N + 
0{N). We provide examples that show that these bounds are tight up to a linear 
term. Finally, we also show that this method can be applied to study the completion 
time of distributed algorithm. 

We believe that the interest of this method is twofold. On the one hand, it 
gives a new insight on the coupon collector by providing a new proof of a more 
general results. We also think that this results could be adapted to more general 
stochastic approximation algorithms and that this could be helpful to understand 
the relationship between the extinction time of stochastic models and the time for 
a fluid approximation of it to get close to extinction. 

Related work. Stochastic approximation algorithm have been introduced in [14] 
for solving root finding problems. Application of these methods are scattered on 
many fields, like economics [■',] or computer science [2]. In all of these works, a first 
step is to show that the stochastic system can be approximated on any finite time 
interval by a fluid approximation ~ e.g. described by a differential equation. Then, 
this approximation is used to derive asymptotic properties such as characterizing 
the limiting dynamics [1] , computing approximation of the steady-state distribution 
[3, 2] or proving stability properties [-5, 7]. 

However, there are few results on the relation between the time for a stochastic 
process to escape a region and the behavior of a fluid approximation of it. In [ ], 
the authors shows that if the time for the differential equation to escape a region 
is finite, then the time for the stochastic system to escape this region converges to 
the same value (Theorem 4.3 of [6]). When the differential equation stays far from 
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the absorbing boundary, the time for the stochastic system to reach its absorbing 
state can be bounded by large-deviation resuhs, as in [ :]. Our case of interest in 
this paper is that the deterministic system converges to a fixed point but does not 
reach this point is finite time while the stochastic process does hit this point in 
finite time. 

Outline of the paper. The rest of the paper is organized as follows. In Section 2, 
we give a formal definition of the problem. Section 3 contains the main results of 
this paper. We show that T^f is bounded by Nt[^ + 0{N) and derive an asymptotic 
development of t]\[. Section 4 establishes a explicit link between Tjv and the average 
completion time of one algorithm. Finally, we show how this can be used to compute 
the completion time of randomized algorithms in Section 5. 



2. Formal description and notations 

Let y be a Markov chain on a finite state space 5 = {0 . . . S'} and let denote P 
its transition matrix. We assume that this chain has an absorbing state, denoted 
0, all the other states being transient. If / denotes the identity matrix, them P — I 
can be written as: 



P-I = 





Q° Q 



where Q is a non-singular matrix such that for all i,j: Qa < 0, Qij > and 
J2j Qij — ^- Q^ ^s ^ vector such that for all i, ^ Qij + Q^ — Q. 

We consider a Markov chain on S^ composed by N copies of the original Markov 
chain. Its state at time t is denoted {Xi{t), . . . XN{t)). The evolution of the Markov 
chain is as follows: 

(1) at each time step, a chain i e {1 . . . N} is picked uniformly at random 

(2) the ith chain Xi{t) changes its state according to the transition matrix P. 
The states of the other chains do not change. 

Our goal is to compute the time for all chains to hit starting from a state 
{xi . . . xn) S S^ . We define T/v this hitting time: 

(1) Tjv =' inf {t : (Xi(i), . . . XM{t)) = (0 . . . 0)} . 

2.1. Notations. For a state x G S, we denote by e^; the line vector that has all 
coordinates equal to except for the xth one which is equal to 1. The vector 1 
denotes the column vector with all coordinates equal to 1. For a line vector a and 
a matrix P, aP denotes the classical matrix product. For example, if P is a 5 x S* 
matrix, then aPl = J2i=i J2j=i o^iPtr 

For each state a; € 5, we denote by W{x) the hitting time of starting from x: 
if y is a Markov chain of transition probability P, then: 

W{x) "^^ E [inf{t : Y{t) = 0} | Y{0) ^ x] . 

Using the vector notation above, we have P (inf{< : Y{t) = 0} > 1 + i | Y{0) = x) = 
e^Q^l for all i G N. Thus, W can also be written W{x) — J^'^o ^xQ'^- 
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3. Hitting time and fluid approximation 

In this part, we bound the expectation of T/y using a deterministic ordinary 
differential equation (ODE) approximation. In particular, we show that E [Tjv] is 
bounded by N times the time ijy for the linear ODE (3) to be at distance 1/A^ 
from its fixed point plus a term of order 0{N). Moreover, this time t^ is of order 
f2(log(-/V)), showing that the term in 0{N) becomes negligible compared to Nt^ 
as N grows. 

3.1. A differential equation approximation. For any state x £ S and any time 
step k, we define the quantity M^ (k) to be the proportion of Markov chains that 
are in state x at time step k: 

1 ^ 

M^ik)^-J2^xm=^^ 

4 = 1 

where lxi{k)=x equals 1 if Xi{k) = x and otherwise. M^ {k) denotes the vector 
of all M^{k) for x € S: M^ (k) — J2xes ^^x {^)^x where e^ denotes a unit vector 
having its xth coordinate equal to 1 and the others 0. The process M^{k) is a 
Markov chain: with probability M^ [k), a chain that is in state i is chosen and goes 
with probability Py in state k. This shows that the expected variation of M^ [k) 
during one time step is: 

E [M^{k + 1) - M^(fc) I Af^(fc)] ^Y.11 Mrik)P^j^{e, e,) 

1 






(2) = ^M^(fc)Q. 

The function f : rrn-^ mQ is called the drift of the system. 

Let us consider the system of differential equation corresponding to the drift: 

(3) m{t) ^ m{t) ■ Q. 

Equation (2) shows that M^ {k) can be described by a stochastic approximation 
with constant step size 1/iV: it corresponds to a Euler discretization of the ODE 

(3) with a random noise U {i.e. such that E [U^ {k + 1) | M^(fc)] = 0) 

(4) M^(fc + 1) = M^(fc) + ^ (/(Af^(fc)) + U'^ik + 1)) . 

Let us call M^ {t) the state of the system when the time has been rescaled by t: 
M^ {t) ~ M^([iiVj). Using classical tools of stochastic approximation (Theorem 1 
of [2] for example), one can show that if M^(0) converges in probability to r7i(0), 
then M'^(t) converges in probability to m{t) uniformly on [0;T]: 

lim sup ||M^(i) — r7i(t)|| = in probability. 

However, when one wants to compute the hitting time of the by M^ (t) , this 
approximation is not accurate enough and leads to overestimated bounds. In the 
following, we will see how to link the hitting time of M^ (t) and the time for TOo(t) 
to be greater than 1 — 1/A^. 
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3.2. Link between the hitting time and the time to reach 1/A^. Let us now 

look at the quantity M^ (t) which is the proportion of Markov chain in state 0. 
The quantity Tjv can be defined as 

Tjv-inf{i:Afo^(i) = l}. 

If m(i) is the solution of the ODE (3), one clearly has liniTv-j-oo 'TT'o(i) = 1 but 
unless if m{t) starts exactly with moit) = 1, the time to reach mo{t) = 1 is infinite: 
inf{t : mo{t) = 1} = +oo. 



Due to the discrete nature of M^ 



(i), M^{t) takes values in {0, i, |, . . . , ^}. 
Thus, when M^ {t) is greater than 1 — 1/A^, it is equal to 1. This suggests to 
introduce ijv, the time for the ODE to be such that mo{t) > 1 — 1/iV: 



(5) 



tN = inf{t : mo{t) > 1 - — }. 



On Figure 1 are reported two simulations for the coupon collector with 2 cards. 
We compare the hitting time tjsi of the stochastic system for A^ = 20 and N — 1000 
with the time t^ for the ODE to reach 1 — 1/N. The time of the stochastic system 
has been accelerated by N. This suggests that tpf is indeed a good estimate of 

Tn/N. 
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(a) Af = 20 
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(b) N = 1000 



Figure 1 . Comparison of the hitting time of the stochastic of the 
system rescaled by 1/A^ and the time for the differential equation to 
reach 1 — XjN for the coupon collector problem with 2 cards. The 
smooth curve represents the differential equation mQ{t), the dotted 
line is the line 1 — 1/A'^ and the curve with the jumps represents 
M^{Nt) for one sample of the simulation for N = 20 or N = 1000. 
For each curve, the hitting time of 1 for the stochastic system is 
close to the hitting time of 1 — 1/A^ for the deterministic ODE. 



Classical stochastic approximation results show that the rate of convergence 
of M^{t) to m{t) is of oder 0{1/^/N). The bound is too loose to guarantee the 
convergence of i^v to T/y. In the next Theorem 1, we use a slightly different approach 
to show that Nt]\[ is indeed a very good approximation of Tjy. 

Theorem 1. Let t^ he defined by Equation (5) with m satisfying the differential 



equation (3) with initial condition m(0) ~ a = N ^ ^^^ 



Then, hitting time 
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Tn o/ (0 ... 0) for the stochastic system composed of the N chains starting from 
[xi . . . x„) satisfies: 



E [Tn] <N[tN + a{I- R^H + 2mSixQj^^ ) = iYijv + 0{N), 



where R is the matrix defined by Ru — and Rij — — ^^ for i ^ j ■ 

Proof. The outline of the proof is as follows. The main idea is to write T/v as 
the maximum of N dependent random variables that correspond to the time for 
each chain to reach its absorbing state. Then we establish a relation between the 
expectation of this maximum and the tail behavior of the marginal distribution of 
each random variable. The marginal distribution for each chain follows a phase- 
type distribution. We show in Lemma 1 that its tail behavior can be approximated 
by the one of a continuous phase- type distribution that leads to the term ijy. 

Let us pick a chain i e {1 . . . N} at random. The distribution of the initial state 
of i is the distribution a. If i was alone, the probability for this chain to be in the 
absorbing state at time step k starting from state i would be (P'^)i,o — ct{I+Q)'^l. 

When considering the system composed by the N Markov chains, the Markov 
chain i makes a transition with probability 1/A^. Thus, the probability for this 
particular chain to be in its absorbing state at time k is a{I+N~^Q)'^l. Therefore, 
the time Ti at which the Markov chain i has hit its absorbing state satisfies: 

(6) p(r>fc) = a(i + lo)^i. 

If Ji . . . ijv denotes a random permutation of {1 . . . N}, then the time for all 
the Markov chains to have hit is T/v = maxi<i,</vT'''. The variables T**^ are 
identically distributed following the law given by Equation (6). However, these 
variables are not independent. 

Using the union bound and the fact that V {Tn > fc) < 1, we have: 



'(Tn >k)< min (l,^P(r^'= > fc) j = min (l,Na{l + ^Q)''! 



Therefore, the expectation of Tn can be bounded by: 

oo oo ^ _. 

E [Tn] ^Y.^'^^^-'^^-Yl ™"^ ( 1' ^"(1 + M^^''^ 



k=l k=l ^ 



2:jY — 1 oo _. 

(7) < E 1+ E Na{l + -Q)''l 



N 

fc— 1 A;— xjv 
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where xn = min{fc G N : a(l+A^"^Q)'=l < 2/7V}. Moreover, using that Efela:„(l- 
N-^Qf = -(1 + N-^QY'^NQ-^, we have 



k=XN 



S 



.7 = 1 ^ 



(8) < max (~Q-i) TV 



2 



= 2max(-g^J 

where the Inequahty (8) conies from the definitions of x^. 
Combining this inequahty and the Equation (7), we get: 



K\Tn] < XN + 2Nma.x(~ 

.7,fe V 



'i,fc 



The quantity xjv is defined by xn = min{fc : a(f + ^Q)^l < 2/N}. We show in 
Lemma f that: 

XN < A^infjt : exp (iQ) < —} + a{I - Ry^l 

= NtN + a{I - R)-^l, 

where i? is a matrix defined by Ra — and Rij — ~^^ for i ^ j- □ 

3.3. Discrete and continuous phase-type distribution. A random variable 
such that ¥{X >t) = aexp(t(5)l is said to have a continuous phase- type distribu- 
tion of parameter (Q, a). Let Y(t) be a Markov chain on S such that the rate of 
transition from i ^ to j ^ i is Qij and the rate of transition from to i 7^ is zero. 
If a is the initial distribution of 1^(0), then the time for Y{.) to reach zero follows 
a phase-type distribution of parameters {Q,a). Similarly, a random variable such 
that P (X > fc) = a(l + Q/N^l is said to have a discrete phase-type distribution 
of parameter (1 + Q/N,a). This corresponds to the time for a discrete Markov 
chain of transition matrix 1 -t- Q/N to reach zero. We refer to [Id], Chapter 2 for 
more definitions and properties of phase-type distributions. 

The next lemma shows the relation between the tail of a continuous phase-type 
distribution of parameter {Q,a) and the tail of a discrete phase- type distribution 
of parameter (1 + Q/N, a). 

Lemma 1. Let xn = minjfc G N : a(l + N^^Q)''! < 2/N} and t^ = Nmm{t G 
M : a exp{Qt)l < 1/A^} be defined as in the proof of Theorem 1. Then: 

XN <N (ijv + a(l - Ry^'i-) , 
where R is a matrix defined by Ra — and Rij — —Qij/Qu for i ^ j . 

Proof. Let us consider the Markov chain ¥{) associated with the continuous phase- 
type distribution of parameter {Q, a). With probability a.;, the Markov chain starts 
in state i. If after k jumps, the Markov chain is in state i, it stays there for a time 
Tki exponentially distributed of parameter —Qu and then jump to a state j ^ i 
with probability —Qij/Qu- Thus, the probability of being in state i after k jumps 
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is [aR )i where R denotes the matrix with Ra — and Rij ~ ~Qij /Qu f^or i ^ j. 
Therefore, if X^ is a continuous phase-type random variable of parameter (a, Q), 
X^ has the same distribution as: 



(9) 



oo S 
k=Q i=l 



where Uki are (dependent) Bernoulh variables of parameter {aR^)i and Tki are 
(independent) exponentially distributed variable of parameter —Qu- 

Similarly, the quantity a{l+N~^Q)'' corresponds to the probability for a discrete 
phase-type random variable to be greater than k and a variable X with discrete 
phase-type distribution of parameter (a, I + Q/N) has the same distribution as: 



oo S 



(10) 



k=0 i=l 



(N) 
ki 



(AT) 



where Uki are the same variable as before and T^^ are independent geometric 
random variables of parameter —Qu/N 



Since T,:.- ' is a geometric random variable of parameter —Qu/N , for all t e M+, 



we have: 



(t^P > tN) 



= (1 
<(1 



N 



[tAl 



" \tN+l 



N ' 
< exp{-Qii(t 



1 

' N 
1 

N 



)) 



where the last inequality comes from the fact that log(l + x) < x. This shows 
that N~^tI:- is less than Tki + jj (for the stochastic order). As pointed out, all 
the T^- and Tki are independent in Equations (9) and (10). Therefore, we can 
assume that N'^^T^- < Tki + j^ almost surely. Using that for any positive random 
variable A^ B and any t e (0; oo) and i e [0; i], we have: 

V {A + B >t) <V ({A> t)\J {B >t - ()) <V {A>t - l) +f {B > tj , 

this shows that for any I: 



C30 S 



EE^^-^r^^^O^ 



\k=0 1=1 



(11) 



< 



(oo S oo S -, \ 

Y^Hu^^Tk^ + HH^^^N-^] 
k^O i^l k^O i^l / 

/ oo S \ / oo S \ 

E E ^fc^^fc^ > i - ^ + p E E ^fc^ ^ ^ 



^fe=0 1=1 



By Markov inequality, P (^^o T.Li ^i 



'k^>Nt] < E 



fc=0 i=l 
S 



Er=oE 



Uk^\/{Ne), 



with E 



EZoEUu, 



ki 



X;fcai?'=l = a(l - R)-^l. This shows that if £ 
a{l — i?)^^l, then the second part of (11) is less than 1/A^. Moreover, iit ^ £ + t 



N, 
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the first part of (11) is less than \/N. This shows that if i > ^at + a(l — i?)^^l, 
then: 

pff:Et/..Tif)>A.t)<A. 

\fc=0 i=l / 

Thus, this shows that xn < N{tN + a(l - -R)"^l). 

D 

3.4. The logarithmic trend. The quantity aexp(Qi)l is equal to one minus the 
cumulative distribution function F{) of a continuous phase- type random variable of 
parameter {a, Q). According to Theorem 2.7.2 of [10], there exist 7 > and /c > 
such that the density of this variable /() satisfies: 

(12) f{t) = -it^ exp(-iyt) + o(i'= exp(-i/i)), 

where —v\s& eigenvalue of Q such that v > Q and fc + 1 > 1 is the multiplicity of 
the eigenvalue —v. 

Equation (12) leads to the logarithmic bound for T/v, expressed by the following 
theorem. 

Theorem 2. Let —v he the eigenvalue of Q with the greatest real part and let 
fc + 1 > 1 denotes its multiplicity. Then, v is real and positive and T/v satisfies: 

E[Tn] < -Nlog{N) + -N\og\ogN + 0{N). 
Proof. By Equation (12), the cumulative distribution function F satisfies: 

/oo 
f{s)ds = 7t*= exp(-j/t) + o{t'' cxp{-iyt)). 

Let sn{x) — i'^^{\og{'yN) + kloglogjN — klogi/ + x). For all fixed x, we have 
sn{x) — tpf + v^^x + o{1). Using that exp(— i/SAr(a;)) = -^(log7A^)~''z^'^ exp(— x), 
the quantity si\f{x)'' exp{—i'SN{x)) is equal to 

v^'' CloghN) + k log log 7iV - fc log z^ + x)'' (log ^Ny^v^ exp(-a;) 

7iV 

(13) ^4.exp(-.)(l+ ^^°g^"gf-^^°g- + - )-^ 

7A' fog 7A' 

The last factor of (13) goes to 1 as TV goes to infinity. Therefore, if x > (or 
X < 0), then (13) is strictly less (or greater) than \/^N if N is large enough. This 
shows that for all e > 0, if A'^ is large enough, we have 

1 - F{sN{-e)) <1/N <l- F{sN{e)). 

This shows that the number t^ such that t > t^ implies aexp{Qt)l < ;jy, is 
equal to: 

(14) tN= ^ (log(7iV) + fcloglogiV-fclog(i.)+o(l)). 

Combining (14) and Theorem 1 concludes the proof of the theorem. D 
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3.5. Application to the coupon collector problem. Let us consider the clas- 
sical coupon collector problem: there are N different types of coupon. At each time 
step, a coupon of type i is picked at random where i is drawn uniformly at random. 
It has been shown in [ ] that the time to collect T coupon of each type is bounded 
by N{\ogN + (T— 1) loglog A^ + 0(1)). In this section, we show that our approach 
allows one to retrieve this result directly. 

1 1 

(a) Collecting T cards of each type (b) Classical coupon collector 

Figure 2. Markov chains used to represent the coupon collector 
problem: the state indicates the number of coupons of that remain 
to be collected. 

Let us consider the Markov chain represented on Figure 2(a). Its state space is 
{0...T}. The initial state is T and for all < i < T: Pj,j_i = 1. The matrix Q 
corresponding to this Markov chain is a T x T matrix that has —1 on its diagonal 
and 1 on its sub-diagonal. On Figure 2(b) is represented the particular case for 

r = 1. 

The ODE corresponding to this system is: 

wt(0 = —mT{t) 

miit) = —mi{t) + mi^i{t) for < i < T 

with mT(0) = 1 and ■m^{t) = for i G {0 . . . T - 1}. 

A direct computation shows that toq (t) is the cumulative distribution function of 
an Erlang variable of parameter (T, 1) {i.e. the sum of T i.i.d. exponential variable 
of parameter 1) which can be written: 

T-l 

mo{t) = 1 - ^ exp(-a::)a;''/fc! 

fc=0 

= 1 - exp(-x)a;^"V(r - 1)! + Oiexp{-x)x'^-^). 

Using Theorem 1, this shows that the time T^ to collect T cards of each type is 
bounded by 

E [Tjv] < log TV + (T - 1) loglogiV + (T + 2)N + 0(1). 

where T + 2 comes from Q^^ = —lj<i and I — R = —Q. 

4. Explicit formula for two particular cases 

Theorem 2 gives a precise idea on the behavior of Tjv in the general case. How- 
ever, the computation of the constants t^, k or 0{N) can be difficult when the 
state space of the original Markov chain is large. In this section, we derive explicit 
formulas for these constants assuming that the hitting time for one Markov chain 
is bounded by T. 

We first show that if the hitting time of the absorbing state is bounded by T 
for all single chain, then T/v is less than N'^T (Theorem 3), which is a loose bound 
in many cases. When the hitting time of the absorbing state is uniformly bounded 
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by T for all initial states x e S, then E[Tn] is bounded by TA^logiV + 0{N) 
(Theorem 4) . At the end of the section, we provide two examples that shows that 
these bounds are tight. 

The results presented in this section remain valid if the state space of the chain 
is countable instead of finite. 

4.1. Unbounded case. If W{x) denotes the expected hitting time of for a single 
Markov chain, then the following results hold. 

Theorem 3. The time Tn such that all the chains have reached is bounded by: 

N 

E[Tn]<nY,W{X,{0)) 

i=l 

In particular, if for all i, W{Xi{0)) < T, then 

E [Tn] < TN^. 

Proof. For alH e {1 . . . TV}, let us call Ri the time at which the Markov chain i 
reaches 0: Ri = infjt : Xi{t) = 0}. It should be clear that Tjy — maxi<j<jv -Rj < 
X]i=i ^i- Moreover, the hitting time for just one Markov chain when it performs 
one transition at each time is W{Xi{0)). As the probability for a Markov chain to 
perform one transition during one time step is 1/A^, we have E [Ri] = NW{Xi{Q)). 

a 

This result seems to be in contradiction with Theorem 2 that shows that if 
we fix a Markov chain, the expected hitting time of a system composed of N of 
these Markov chains is bounded by 0{N log N). However, the constant hidden in 
0{N\ogN) depends on the Markov chain and the trend in A^logA^ is only valid 
when the number N goes to infinity while the bound TN'^ only depends on T. At 
the end of the section, we provide an example that shows that this bound is tight. 
The Markov chain used for this example depends on N. 

4.2. Uniformly bounded case sup^.^^ W{x) < T. On Figure 3(a), we present a 
Markov chain that the bound of the previous theorem is tight. This chain has a very 
particular shape: starting from the initial state, the hitting time of the absorbing 
state is 1 with a probability 1 — 1/N'^. With probability l/N'^, the chain jumps into 
a state from which is takes N'^{T — 1) steps to hit 0. This later causes the hitting 
time to be large when multiple chain. 

In this section, we show that if there are no such problematic states, the bound 
on T^v can be improved dramatically. More precisely, we show that if the hitting 
time of is bounded by T independently of the initial state - sup^.^^ W{x) < T, 
then Tn is of order TN log N. 

Theorem 4. // the hitting time for one chain is uniformly bounded by T (i.e. 
supj.g5 W{x) < T), then the time T^ such that all the chains have reached satis- 
fies: 

E [Tn] < TNlogN + 2NT + 1 

Proof. Let Tt denotes the filtration associated to the process X{t) and let us define 
the potential of the system at time i, $t by: 

1 ^ 
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T/v is the time at which all Xi{t) are equal to and can be written T/v = inf{t : 
$(t) — 0}. In the following, we first show that the time for <i>(i) to be lower than 1 
is less than NT log N + NT + 1 using [ ' ■"'•] . Then, we use Theorem 3 to bound the 
remaining time by NT. 

We say that a Markov chain is active if it did not reach 0. If an active Markov 
chain is picked in step 1, then the potential will decrease in expectation by l/T. 
Let a{t) denotes the number of active Markov chains at time t (i.e. a{t) = 
J2i=i '^Xi(t)^o)- The probability of picking an active Markov chain is a{t)/N. 
Therefore, the expected decrease of the potential between time t and i + 1 is: 

(15) Emt+i)-^t)\^,]<-^. 

By hypothesis, sup^g^ W{x) < T. Thus, an active processor contributes at most 1 
to the potential and we have ^{t) < a{t). Combining this with (15), we get: 

(16) Em+l)\Tt]<nt)(^l-^y 

Because of Equation (16), our potential function satisfies the hypothesis of The- 
orem 1 of [1 "] with m ~ 1 and h{r) ~ 1 — 1/{NT). According to this theorem, we 
have: 

inf{t : $(t) < 1} < A(log($(0)) + 1) + 1. 

where A = -l/log(l - 1/{TN)) < TN and $(0) < N. 

By Theorem 3, when Tpf is less than 1, J^t^i^tii)) < T. Therefore, the 
remaining time to hit is bounded by NT. 

O 

4.3. Comparison with previous bounds and tightness. Theorems 3 and 4 
need stronger assumptions than Theorems 1 and 2 and are often less precise. How- 
ever, their main advantage is to give explicit formulas for the hitting time, even if 
computing the time tpj or the eigenvalue of the individual Markov chains is hard. 
This fact is important in practical situation where the Markov chains often have a 
complicated geometry. This is the case for the example of [4] presented in the next 
Section 5. 

The loss of precision of these bounds are well illustrated by the coupon collector 
problem. Consider the Markov chain of Figure 2(a) that corresponds to the problem 
of collecting T cards of each type. The hitting time of from any state is clearly 
bounded by T. Therefore, using Theorem 4, one has E [Tn] < TN(\ogN + 2) + 1. 
This bounds is worse by a factor T compared with the bound obtained by the ODE 
approach which was 7V(log A^ -I- (T — 1) log log N) + 0{N). This is explained by the 
fact that Theorem 4 does not take into account the particular shape of the Markov 
chain of Figure 2(a): the bound of Equation (16) neglects the fact the hitting time 
starting from state {0 ... T — 1} is strictly less than T. 

4.3.1. Tightness of the bounds of Theorems 3 and 4- A Markov chain that shows 
the tightness of the bound of Theorem 3 is represented on Figure 3(a). The chain 
has N + 2 states, denoted {0, . . . ,N'^{T~ l)}U{i}. Its initial state is i. From i, the 
chain goes with probability 1/A^^ to state N'^{T — 1) and with probability 1/N'^ in 
state 0. From any state x G {1 . . . N'^{T — 1)}, the chain goes to state x ~ 1 with 
probability 1. For any state x e {0 . . . N^{T — 1)}, the expected hitting time of 
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(a) Example for Theorem 3 (b) Example for Theorem 4 



Figure 3. Two Markov chains used to show that the bounds of 
Theorems 4 and 3 are tight. 

the state is W{x) = x. When starting in i, the expected hitting time of is T. 
Therefore, Theorem 3 shows that the expected hitting time of (0 ... 0) for a system 
composed of A^ of these chains starting in {i . . A) is bounded by TN"^ . 

Let us compute a lower bound on the hitting time of (0 ... 0) starting from 
(i . . . i). With probabihty 1 — (1 — 1/7V^)^, there will be at least one chain that 
needs N'^{T — 1) transitions to converges. At each time step, this chain makes a 
transition with probability 1/A^. Thus, in average this chain will take (T — \)N^ 
time steps to converge. Since this happens with probability 1 — (1 — l/N"^)^ , a 
lower bound for the hitting time of (0 ... 0) starting from everyone in state i is 

E [Tjv] > N^iT - 1)(1 - (1 - 1/N^)^) = (T - 1){N^ + o(l)). 

This shows that the bound of Theorem 3 is almost tight up to an additive term of 

To show that the bound of Theorem 4 cannot be improved much without further 
assumption, let us consider the Markov chain represented on Figure 3(b): it has two 
states and the probability of going to 1 to is 1/T. The expected hitting time of 
state starting from state is T. Theorem 4 implies that Tn < NT {log A^ + 2) + 1. 
The exact value for Tn is NTj^f^i J~^ ~ NT log N + jNT + o{N) where 7 w .57 
is the Euler-Mascheroni constant. This is close to our theoretical bound up to an 
additive term of (2 - j)NT. 

5. Computing completion time of randomized algorithms 

In this section, we show how these results can be applied to study the completion 
time of randomized algorithms and show how this can be used to design efficient 
distributed protocols. 

5.1. Completion time of randomized algorithm. One motivation for this work 
comes from the study of the time for a set of N distributed randomized algorithm 
to all finish, in a scenario similar to [ ]. 

Let us consider that we want to solve a resource allocation problem among 
a population of agent. We assume that we have a randomized algorithm that 
converges to a stable allocation of the resource that is efficient but not fair among 
different agents. The final allocation might depend on the random choices done 
by the algorithm and each allocation favors a different group of agents. In order 
to improve the fairness of the equilibrium, we consider the following scenario. We 
execute N independent copies of the algorithm. At each time step, we do a step 
of computation of one algorithm taken at random among the N algorithms. After 
some time, the N algorithms will have reached their stable allocation Si . . . S^. 
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Since each allocation is efficient, the resulting allocation will also be efficient but 
the resulting allocation will be more fair since at each time step, one allocation is 
picked at random among Si . . . Sn . 

If the original algorithm uses bounded memory, it can be represented by a Markov 
chain with a finite state space. After some time, the Markov chain will reach an 
absorbing state representing the fact that the algorithm reached a stable allocation. 
The resulting algorithm can be represented by N independent Markov chains. At 
each time step, one Markov chain is picked at random and performs one transition. 
Our framework, and in particular Theorems 4 and 3 can be used to compute the 
time to reach the final allocation if we know the time taken by a single algorithm 
to converge. 

5.2. Correlated equilibria and distributed protocols. These ideas are ap- 
plied in [4] to design a distributed algorithm that converges to a fair and efficient 
allocation of wireless radio channel to a set of user. 

Their scenario is the following. There are U users that want to share C wireless 
channels. The time is slotted and at each time slot, each user can transmit data 
on one channel. It two or more users are transmitting on the same channel at the 
same time, there are interferences and no data are received. The only information 
available to a user before transmitting is whether a given channel was used by 
one or more users at the previous time slot. Authors of [ ] proposed a distributed 
randomized algorithm that converges to a constant assignment of the C channels 
to C of these U users while the others U — C users do not transmit at all. This 
algorithm guarantees a 100% of utilization of the channels but is unfair since U — C 
users are not transmitting at all. The time of convergence of the algorithm can be 
bounded by some constant T. 

In order to improve the fairness of their algorithm, they introduced a centralized 
entity that sends a correlation signals. At each time step i, this entity sends a 
signal n{t) S {!,... , A^} where A^ is a predefined constant (to be chosen by the 
entity). The signal n(t) is picked uniformly at random each time. The users keep 
N copies of the previous randomized algorithm. At time t, the users apply the 
algorithm number n{t). Even if this algorithm is no more completely distributed, it 
is still scalable: the centralized entity has to broadcast a signal n{t) at time t but it 
does not have to gather any information from the agents. Moreover, this algorithm 
improves the fairness of the initial algorithm. As each algorithm is independent, if 
N is large, each user will be assigned in average to C /U channels. The more N is 
large, the more fair will be the allocation. However, a large N slows the convergence 
of the algorithm. An accurate bound on the time of convergence of this algorithm 
allows one to choose the right compromise between the speed of convergence and 
the performance of the algorithm. 

As the convergence time of each copy of the algorithm is bounded in expecta- 
tion by some T, this model satisfies the hypothesis of Theorem 4. Therefore, the 
convergence time of the whole algorithm is bounded by NT log N + 2NT + 1. 
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